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ABSTRACT 



A method and apparatus that enables a user to store or 
indicate event information while watching a television 
broadcast is disclosed. The event information is transmitted 
to a server, preferably via the Internet, either in batches or in 
real-time. Based on the event information, the server deter- 
mines which program the user was watching when the event 
information was stored or indicated, and the time within that 
program. Based on the determined program and time, the 
server determines an assortment of goods and services that 
were displayed on the user's television when the event 
information was stored. This assortment is presented to the 
user, preferably via the Internet. 

19 Claims, 22 Drawing Sheets 
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INTEGRATED TELEVISION AND INTERNET 
INFORMATION SYSTEM 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

This application is a continuation-in-part of application 
No. 09/252,071, filed Feb. 18, 1999, which is incorporated 
herein by reference. 

BACKGROUND OF THE INVENTION 

This invention relates to the field of facilitating commerce 
by providing access to information about goods and services 
that are displayed in television broadcasts. 

Using television commercials to promote products is a 
widespread practice. Typically, viewers watch certain pro- 
grams on television because they are interested in those 
programs. Television stations capitalize on this situation by 
broadcasting commercials interspersed with the program 
itself. These commercials typically try to convince the 
viewer to buy a particular product. As used herein, the term 
"product" refers to both goods and services. 

Interspersing commercials within television programs, 
however, has some significant drawbacks. For example, 
from the perspective of the viewer, television commercials 
are often perceived as annoying interruptions to the program 
that the viewer wishes to watch. Even when a viewer is 
interested in a particular product in a commercial, the viewer 
may prefer to find out about it after he has finished watching 
the program. This cannot be accomplished with conven- 
tional television broadcasts. 

Another drawback, from the viewer's perspective, is that 
the viewer has no control over the subject matter of the 
commercials that he will see, since the content of traditional 
commercials is determined solely by the advertisers. As a 
result, many viewers may not be interested in the commer- 
cials that they see. 

In contrast, since the viewer selects the program that he is 
watching, the viewer is presumably interested in the con- 
tents of that program. During a conventional broadcast of the 
program itself, however, the viewer has no way to obtain 
information about any products that he sees in the program 
(e.g., a certain jacket being worn by an actor in a TV sitcom). 

Traditional commercials also have drawbacks from the 
broadcaster's perspective. First, because commercials only 
take up a relatively small fraction of the total time of a 
broadcast, the amount of air time that broadcasters can sell 
is limited. And if a broadcaster attempts to overcome this 
limitation by increasing the amount of commercials, the 
broadcaster risks losing its viewers, because they may 
switch to other stations, only to return once the commercial 
is over. Hence, no benefit is obtained for the advertiser, the 
broadcaster, and the consumer. 

The Internet is another conventional forum in which 
products are advertised. These advertisements typically take 
the form of advertisements displayed on a website and 
generally appear on the screen simultaneously with the 
desired information, in a distinct section of the display. 
Unlike traditional television commercials, such advertise- 
ments do not preclude the user from viewing other infor- 
mation. If the user is interested in the Internet advertisement, 
the user can click on the advertisement and proceed accord- 
ingly. If on the other hand, the user is not interested, the user 
simply ignores the advertisement and continues obtaining 
the information in which he is interested. Internet advertise- 
ments are therefore less intrusive than television commer- 
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cials. Internet advertisements also provide an added benefit, 
in that, once an interested customer has been found, a sale 
transaction can be facilitated on the spot. 

Despite these advantages, however, product commercial - 
s ization using the Internet can only take place when the 
relevant users are actually logged on to the Internet. Even 
among families with access to the Internet, however, the 
total amount of time spent logged on to the Internet in this 
country is dwarfed by the total amount of time spent 
10 watching television. As a result, the marketing power of 
Internet advertisements is still relatively insignificant in 
comparison to the marketing power of television commer- 
cials. 

A commercialization system that does not suffer from the 
15 drawbacks described above would be desirable to both 
viewers and broadcasters. Until now, however, no such 
systems have been implemented. 

In the past, attempts have been made to combine televi- 
^ sion and Internet commercialization. The simplest of these 
combinations is displaying a URL (uniform resource 
locator), or "address," of a website on the television during 
a traditional television commercial. Tliis system is problem- 
atic for both the viewer and the advertiser, because the 
M viewer may not have a pen handy to write the URL down. 
And even in cases where the user does write the URL down, 
the paper may not be handy the next time the user logs on 
to the Internet. Either of these scenarios would result in a lost 
opportunity to promote a sale, and a lost opportunity for a 
30 consumer to obtain a product in which he is interested. 
Another existing combined television/Internet system is 
WebTV. WebTV allows its users to view a website on a 
traditional television display. Recent innovations have even 
allowed a website to be accessed without interrupting the 
35 television signal, using a picture in picture format. But in 
WebTV, the Internet and television-viewing sections are 
largely independent, and there is no interaction between the 
URL accessed and the television program being viewed. 
Finding information about products that appear within a 
40 show must be accomplished using traditional search meth- 
ods. 

Yet another combined television/Internet system is 
described in U.S. Pat. No. 5,778,181. The '181 patent 
describes transmitting URLs during the vertical blanking 
45 interval of a television broadcasting signal. These URLs are 
extracted at the viewer's home, and the associated web 
pages are fetched (via the Internet) while the viewer is still 
watching his television. In the '181 patent, however, the 
viewer is merely fed two streams of information, with one 
50 displayed on the television, and the other displayed on a 
computer monitor. As a result, many of the disadvantages of 
traditional television viewing remain. In addition, the user 
must divert bis attention away from the program that he is 
watching in order to obtain the information being displayed 
55 on the Internet. 

SUMMARY OF THE INVENTION 

The present invention advantageously overcomes many 
of the aforementioned disadvantages and provides an inte- 
60 grated television/Internet commercialization system. This 
system enables a user to watch television in a traditional 
manner, but also provides the user with an opportunity to 
indicate interest in things that are being displayed on the 
television. This indication could be made, for example, by 
65 pressing a button on a customized handheld remote control. 
In some preferred embodiments, the system stores these 
indications. After the user has finished watching the program 
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(or at such other time, as the user may desire), these apparatus includes a plurality of switches, an infrared light 

indications are transferred to a remote server, which presents transmitter, and a radio frequency transmitter. In response to 

to the user information about the products that were being an actuation of a given one of the plurality of switches, (a) 

displayed at the time the user made each indication. In other mc radio frequency transmitter transmits information iden- 

preferred embodiments, the system forwards the indications 5 tifying the given one of the plurality of switches and (b) the 

to a remote server that notes the time of arrival for each infrared light transmitter generates a sequence of infrared 

indication. The server then presents to the user information U gh t p U i&es corresponding to the given one of the plurality 

about the products that were being displayed at the time the of switches. 

user made each indication. According to yet another aspect of the present invention, 
According to one aspect of the present invention, a 10 a remote control apparatus is provided. The remote control 

method of commercializing products that are present in a apparatus includes a plurality of switches, an infrared light ' 

television broadcast of a program is provided. The method transmitter, and a radio frequency transceiver. In response to 

includes the steps of inputting product information that m actuatioD of a gi vea one 0 f tne plurality of switches, the 

identifies a plurality of products and associated times of frequency transceiver transmits information identify- 

presence in the program, inputting skew information that 15 ^ g ^ given one of uie plurality of switches. In response to 

identifies a correspondence between an actual time of broad- & gigna] received by ^ ra di 0 frequency transceiver, the 

cast and a relative time within the program, and inputting a infrared light transmitter transmits a sequence of infrared 

signal that was generated in response to an indication of ^ pulscg corresponding to the received signal, 

interest made by a user. The method also includes the steps ^ ^ ^ ^ mes ^ advanlages 0 f the 

of determining a time-of-interest based on when the signal w ^ inv ention will be apparent from the following 

arrived, identifying a specific portion of me program that V . description of preferred embodiments. The detailed 

was being broadcast at the time of interest based on the time fa £ be ^ . q connecUoD ^ tne accompa . 

of interest and the inputted skew information determining * ^ wfaich: 

an assortment of products that were present in the television ' & ^ 

broadcast at the time of interest, and presenting the deter- 25 BRIEF DESCRIPTION OF THE DRAWINGS 

mined assortment of products to the user. rf of a first 

According to another aspect of the V™*"^** ^ embodiment of the present invention, 

method of commercializing products that are present in F J.w.^hpW^mntPnfiiwfim 

multiple simultaneous television broadcasts of programs is FIG. 2 is a block diagram of a handheld remote of the first 

provided. The method includes the steps of inputting, for 30 preferred embodiment. 

each of the programs, product information that identifies FIG. 3 is an example of a user's TV setup which can be 

products which are present in the respective program and used with the present invention. 

associated times of presence, inputting skew information FIG. 4 is a table specifying the user's TV setup, 

that identifies a correspondence between an actual time of piQ. 5 is a table specifying the channel setting for each 

broadcast and a relative time within the respective program, 35 tuner device. 

inputting an event signal that was generated in response to pj G 6 is a table of evenl var i a bles, which is stored in the 

an indication of interest made by a user, and inputting a handheld remote in the first preferred embodiment, 

channel-change signal that was generated in response to a FIG 7 is a table of time stamps, which is also stored in the 

channel change command made by the user. The method handheld remote m ^ first preferred embodiment, 

also includes the steps of determining a time-of-interest 40 tion ^ 

based on when the event signal arrived, determining a ri °- , , " v £ B , . \,. 

channel baled on the channAange signal, identifying a handheld remote of the first preferred embodiment, 

specific portion of the program that was being watched by FIG. 9 is a flowchart depicUng processing ; of umversal 

the user based on the time-of-interest, the determined remote keys in the handheld remote of the first preferred 

channel, and the inputted skew information, determining an 45 embodiment. 

assortment of products that were present in that program at FIG. 10 is a block diagram of a home computer setup used 

the time-of-interest, and presenting the determined assort- with the first preferred embodiment, 

ment of products to the user. FIG. 11 is a flowchart depicting the transmission of time 

According to another aspect of the present invention, a stamps from the handheld remote in the first preferred 

method of commercializing products that are present in 50 embodiment. 

multiple simultaneous television broadcasts of programs is pjG. 12 is a flowchart depicting the reception of time 

provided. The method includes the steps of inputting, for stamps by the home computer in the first preferred embodi- 

each of the programs, broadcast and product information ment. 

that identifies products which are present in the broadcasts pj G 13 ^ a flowchart depicting the processing of time 

and times of presence for each of the products, inputting an 55 stam ps by the home computer in the first preferred embodi- 

event signal that was generated in response to an indication ment 

of interest made by a user, and inputting a channel-change piG u Ulustrates mapp i n g broadcast time into program 

signal that was generated in response to a channel change ^ . ^ ^ 

command made by the user. The method also includes the indicating which channel is used by each 

steps of determining an assortment of products that were 60 idef 

present m the program that was bemg watched at the time U&1 r . 

the user made the indication of interest based on the event FIG. 16Ais a table that stores information relating to the 

signal, the channel-change signal, and the inputted broadcast programs supported by the system. 

and product information, and presenting the determined FIG. 16B is a table reporting the broadcasting of a 

assortment of products to the user. 65 program for a particular network. 

According to yet another aspect of the present invention, FIG. 17 is a flowchart depicting processing that occurs in 

a remote control apparatus is provided. The remote control the central server. 
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FIG. 18 is a flowchart depicting the creation of the Once the user 101 finishes watching television 102, he 

Product Display Table. takes the handheld remote 105 over to his home computer 

FIG. 19 is a table indicating when each product appears 106. Preferably both the remote and the computer arc 

. - • _ equipped with displays and IR (infrared) transceivers. The 

and disappears in given program. ^PP ^ ^ / remote's IR transceiver at the IR 

FIG. 20 is a table of product information. transceiver on the computer, and presses the transfer button 

FIG. 21 is a flowchart depicting the conversion of time on me remote to establish a communication session with the 

stamps to products. home computer. Set-up of this communication session may 

FIG. 22 depicts an alternative embodiment that is imple- be facilitated by appropriate prompts to the user 101 on the 

mented in an application space on a computer. displays on the remote 105 and the home computer 106. 

FIG. 23 is a block diagram of a handheld remote used in 10 During the communication session, information pertaining 

the second and third preferred embodiments. to each press of the event button on the remote is transferred 

FIG. 24 is a flowchart depicting the operation of the ^ the home computer. Alternatively, in place of the IR 

handheld remote of the second and third preferred embodi- communication described herein, other types of communi- 

mcnts cation links including, for example, radio frequency and 

FIG. 25 is a flowchart depicting how received key-presses " X - 10 , communication, may be used, 

are processed in the second and third preferred embodi- After the event information has been transferred from the 

men jg handheld remote 105 into the home computer 106, the home 

computer 106 establishes a connection with the central 

DETAILED DESCRIPTION OF THE server iQ7 via the Internet, and sends the event information 

PREFERRED EMBODIMENTS 20 to the server. 

FIG. 1 is a block diagram of a first preferred embodiment Meanwhile, the server 107 has previously inputted prod- 
in accordance with the present invention. In this uct information that identifies products which are present in 
embodiment, television programs are broadcast from the various programs, and times of presence within the pro- 
broadcaster 104 (e.g., a television station) to the user's home ^ grams for each of the products. This information can come, 
in any conventional manner, including for example, broad- for example, from a database created by the producers of a 
casts via a cable service 108. Alternatively, other types of program. Alternatively, this information could be inputted 
broadcasts can be used, including, for example, satellite and by a person watching the program who has noted the 
ground based antennas transmissions (not shown). Tradi- products in the program and the portion of the program in 
tional television commercials may be interspersed with the 3Q which those products appear. Numerous other scenarios can 
programs and broadcast over the same medium in a con- be readily envisioned. 

ventional manner. In addition, the server 107 has previously inputted skew 
The broadcast signals are distributed within the user's information, described in greater detail below, which indi- 
home in any conventional manner. In the illustrated setup, cates when each segment of a program is actually broadcast, 
the cable signal is provided directly to the television 102, 35 The server uses this skew information to map "broadcast 
and also indirectly, via the cable box 103. Notably, no time" (which is an actual time, such as Jan. 2, 1999, 8:05:02 
modifications to the conventional broadcasting equipment PM) into "program time" (which is a relative time within the 
are required, and no special equipment is required by uncut program, such as 56 minutes and 22 seconds from the 
preexisting customers who are not using the present inven- start of a particular program). Preferably, the skew informa- 
tion. 40 tion will arrive from a cooperating broadcaster that sends the 
Preferably, the system is implemented using a customized skew information to the server while the program is being 
handheld remote control unit 105 which performs all con- broadcast. 

ventional universal remote functions. These functions are The server 107 then determines an assortment of products 

well known and include, for example, selecting a signal that were displayed on the user's television 102 when the 

source, changing the channel on the currently selected signal 45 user pressed the event button on the remote 105, for each 

source, and changing the volume on the TV set. Each of press of the event button. This is accomplished by referenc- 

these functions is preferably implemented using conven- ing a previously inputted product data base that indicates 

tional buttons on the handheld remote. which products appear in the program being watched, and 

In addition to these conventional buttons, the handheld the times that they appear (measured in program time), 

remote unit 105 of this embodiment preferably includes two 50 The central server 107 then sends information about this 

additional buttons: an event button and a transfer button. assortment of products back to the home computer 106 via 

From the user's perspective, the experience of watching the Internet. This can be accomplished, for example, by 

television 102 starts out the same as when he uses a sending a web page or database to the home computer 106. 

conventional universal remote. For example, he can use the The product assortment is then presented to the user 101 on 

remote 105 to change channels or adjust the volume at any 55 the display of home computer 106. The product assortment 

time. Whenever the user 101 sees something on his televi- can be presented to the user in any number of ways. For 

sion 102 in which he is interested, the user presses the event example, a set of windows can be used, with one window 

button on the handheld remote 105, and continues to watch representing each product in the assortment. Alternatively, a 

the program. Thus, the only change (as compares to tradi- still image or a video clip of the selected moment of the 

tional television watching) is the occasional pressing of the 60 program may be displayed. Numerous other alternatives 

event button, which is a minimal interruption. presentation approaches can be readily envisioned. 

Each time the user 101 presses the event button, the This arrangement is advantageous to broadcasters 

remote 105 stores event information. Preferably, this event because it expands commercialization into the time that the 

information includes the time that the event button was television program itself is being broadcast. It can even be 

pressed, as well as the signal source and channel being 65 used to commercialize products that appear during the 

watched at that time. This is described below in greater commercials themselves (such as the shoes being worn by 

detail. the spokesperson in a commercial for a car). 
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It is also advantageous to television viewers because it 
enables them to obtain information on and purchase prod- 
ucts which attracted their interest. 

FIG. 2 is block diagram of the handheld remote unit of the 
first preferred embodiment. Preferably, it includes a proces- 5 
sor 201 that is connected to RAM (Random Access 
Memory) 202, ROM (Read Only Memory) 203, and an I/O 
adapter 204 via a system bus. These components and their 
interconnections are well-known, as evidenced by the large 
number of universal remote controls available today. Most i° 
preferably, all of these components are integrated into a 
single integrated circuit. 

Of course, various equivalents for each of these compo- 
nents may be substituted for the respective component. 
Examples of such substitutions include using hardwired 15 
logic in place of a processor running a program out of ROM, 
or using a RAM with a battery backup in place of the ROM. 
Numerous other modifications will be apparent to persons 
skilled in the relevant art. 

20 

The I/O adapter 204 enables the processor 201 to deter- 
mine which keys on the keypad 205 are being pressed, in any 
conventional manner. In addition, the I/O adapter 204 
enables the processor 201 to communicate with an IR 
transceiver 207. The Vishay Telefunken TFDT5500 is an 
example of a suitable IR transceiver 207. The I/O adapter 
204 also enables the processor 201 to write to the liquid 
crystal display 206, also in any conventional manner. 

The handheld remote of this embodiment performs three 
distinct functions: control of the devices in the user's home 3Q 
(e.g., the television 102 and the cable box 103 shown in FIG. 
1), recording of events, and communication with the home 
computer 106 (also shown in FIG. 1), recording of events, 
and communication with the home computer 106 (also 
shown in FIG. 1). 35 

The control mode is used to send appropriate IR control 
codes to various devices in the users home, as with conven- 
tional universal remote controls. These control codes could 
include, for example, changing the volume on the TV set, or 
changing the channel on the cable box. Communicating with 40 
such devices via IR commands is well-known, as exempli- 
fied by commercially available universal remote control 
devices. In this mode, the IR transceiver 207 operates as a 
transmit only device, and the processor 201 controls the IR 
transceiver 207 by sending signals to the I/O adapter 204. 45 

The event recording mode is used by a television viewer 
to indicate that he is interested in a product that appears on 
the television. Preferably, this is implemented using a dedi- 
cated button located on the keypad 205, which the viewer 
presses to indicate that he is interested in a product that 50 
appears on the television. This button is referred to herein as 
the "event" button, which is not implemented on conven- 
tional universal remotes. Preferably, when a key-press for 
the event button is detected, the processor 201 in the remote 
stores appropriate information in memory in tables set up in 55 
RAM 202. These tables are explained in greater detail 
below. 

In the communication mode, the remote sends the event 
information, which was stored in response to presses of the 
event button, to the computer 106. Preferably, this is imple- 60 
mented using a second dedicated button located on the 
keypad 205, which the viewer presses when he wants to 
transfer information from the remote 105 into the computer 
106 (for subsequent uploading to the server 107). This 
button is referred to herein as the "transmit" button, which 65 
is also not implemented on conventional universal remotes. 
When a key-press for the transfer button is detected, the 
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processor 201 initiates communication with the home com- 
puter 106, via the IR transceiver 207. It is envisioned that 
communication with the user's home computer will usually 
take place after the user has finished watching television, 
although this is not mandatory. 

In this mode, the IR transceiver 207 in the handheld 
remote 105 is used to communicate with the home computer 
106. While it is only necessary to transfer information in one 
direction (from the remote to the computer), the system is 
preferably implemented using two-way communication, to 
facilitate handshaking and to ensure that the data is trans- 
ferred properly. Communication between home computers 
and other devices using IR transceivers is also well known, 
as evidenced by conventional IR technology based cordless 
computer keyboards, communications between handheld 
personal organizers, and the IRDA standard for such com- 
munications. 

A single IR transceiver may be used for both the control 
mode and the communication mode, as explained above. 
Alternatively, the IR transceiver (e.g., an IRDA compliant 
transceiver) may be used only for communication with the 
computer. An independent IR transmitter or transceiver 
would then be used for controlling the entertainment 
devices. 

In the embodiment described above, the device control 
functions, the event button, and the transmit button are 
implemented on a handheld remote. Alternative embodi- 
ments can be readily envisioned, including, for example, 
implementing all these functions on a home computer (or 
combined television home computer), and actuating them by 
pressing keys on the keyboard, by mouse clicks, or by voice 
control. 

FIG. 3 depicts one of many possible cable television 
setups that could appear in a user's home, which will be used 
as an example to explain the present invention. Of course, 
the invention may be practiced with other setups, including, 
for example, setups that receive broadcasts from a land 
based antenna broadcasting VHF signals, a satellite, a Local 
Area Network, or the Internet. In the depicted configuration, 
a cable service 108 is hooked up directly to the "cable" input 
of television 102. The cable service is also routed to a cable 
set top box 103, which in turn sends a signal to the "aux" 
input of the television set 102. With this setup, the user is 
free to select either the cable input (using the tuner in the 
television) or the aux input (using the tuner in the cable box) 
for display on the television 102. 

Preferably, the handheld remote can control selection of 
the signal source that is displayed on the television 102. In 
addition, the handheld remote can control the channel to 
which the television is set, as well as the channel for any 
other tuner device (e.g., the cable box 103) in the system. 
This is preferably accomplished by issuing IR remote com- 
mands. In order to control these devices, the handheld 
remote must be initialized to know which brands of equip- 
ment are installed in the user's home, so that it can send out 
the appropriate IR commands for that equipment. For 
example, if the television set 102 was made by Sony, the 
remote must know this in order to issue the appropriate IR 
commands to control the television 102, 

Initialization of the remote is preferably accomplished via 
IR commands that are transmitted to the remote from the 
home computer via IR transceivers or via a cable. 
Alternatively, other means for initializing the remote may be 
used, including, but not limited to, providing a remote that 
is pre-initialized at the factory to match a user's video 
equipment setup, or providing a printed table that lists codes 
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for each manufacturer, with instructions asking the user to from the tuner device. If a suitable acknowledgment does 

enter these codes using the buttons on the handheld remote. not arrive, the handheld remote can re-issue the command 

The initialization information that specifies the user's *e acknowledgment is received, thereby correcting the 

system setup is preferably stored in a file on the home channel sync error. 

computer in a service and device configuration table 250 5 When a bidirectional interface is not available, it may be 

(hereinafter the SDC table, shown in FIG. 4). Each row of possible to reduce the odds of channel sync errors by always 

this table relates to one device that can display information using IR commands that access channels directly (e.g., a 

on the user's television. The table includes information command that instructs the tuner to go directly to channel 5) 

specifying an input of the television, the device used to tune as opposed to IR commands that increment or decrement the 

to a particular frequency, the manufacturer of that device, the 10 channel The appropriate direct channel-access commands 

service name, and a service ID. Preferably, this table is are generated even when the user presses a channel-up or 

stored in the home computer and created using a program channel-down key on the handheld remote 105. While some 

that lets the user enter information defining his video equip- approaches for dealing with channel sync errors are dis- 

ment setup, or created in the factory from user specified cussed below, the bulk of the discussion that follows 

information. 15 assumes that a channel sync error has not occurred. 

In the illustrated example, the second row of the SDC FIG. 6 illustrates an event variables table 270. This table 

table 250 indicates that to display a station arriving via the includes fields for the date, the time, and the current display 

cable box, the television is set to select the aux input, that the device. Keeping track of the date and time may be accom- 

cable service providing signals to the cable box is Manhattan plished by any number of conventional methods including, 

Cable, that the service ID for Manhattan Cable is 3888, and 20 for example, using a built-in timekeeping feature of the 

that the cable box is made by General Instrument. The third processor in the handheld remote unit, or software that runs 

row of the SDC table contains corresponding information on that processor. The table 270 also includes a field, 

for signals arriving directly at the television's cable input, preferably stored in RAM, which tracks the current display 

without passing through the cable box. In that case, the device. In the illustrated example, this field would be set to 

station is selected using the built-in tuner on the Sony 25 a first value when the television's cable input is selected, and 

television. In this example, both rows have the same service another value when the television's aux input is selected. 

ID, because both the cable box and the cable input of the The handheld remote keeps this field current by updating it 

television are fed by the same cable service. When a single after issuing any command which changes the signal source 

user receives service from multiple providers (e.g., cable and that is being displayed on the television. Preferably, when 

satellite), the SDC table would list a different service ID for 30 two or more images are displayed on the television, as with 

each provider. picture-in-picture televisions, the largest display will be 

FIG. 5 shows an example of a tuner device current state tracked, 

table 260, which is used by the remote in this embodiment When the event button on the handheld remote is pressed, 

to keep track of the channel to which each tuner device is 35 information from the tuner device current state table 260 and 

tuned. Preferably, this table is also stored in RAM in the the event variables table 270 are extracted and stored in a 

remote. In the illustrated example, this table stores informa- time stamp table 280, shown in FIG. 7. Each time the event 

tion indicating that the television tuner is set to channel 5, button is pressed, an additional entry is added to the time 

and that the cable box is set to channel 4. Because the stamp table 280. These entries are referred to herein as "time 

handheld remote is used as a universal remote to control all 4Q stamps" or "markers". The example illustrated in FIG. 7 

the television related devices in the user's home, it is simple includes four such time stamps. The number of time stamps 

for the handheld remote to keep track of the particular mat can be stored in the time stamp table 280 may be limited 

channel to which each of the tuner devices is tuned. This can at a preset number, or may be determined by the amount of 

be accomplished, for example, by updating an internal available RAM available in the handheld remote. Preferably, 

register after each channel-changing command is issued to 45 time stamps are erased from the time stamp table when they 

any device. are transferred to the home computer. 

When a one-way remote control interface from the hand- FIG. 8 illustrates the functions performed in the handheld 

held remote 105 to the currently selected tuner device is remote, which are preferably performed in software running 

used, and a transmission from the handheld remote 105 does on the processor within the remote. In step S300, the 

not arrive at the tuner (e.g., due to improper aiming of the 50 program waits for a key to be pressed. This can be accom- 

handheld remote 105, or if an object blocks the IR beam), it pushed in any number of ways including, for example, 

is possible that the channel setting in the handheld remote receiving an interrupt from a keyboard interface or by 

105 may not match the actual channel setting of the tuner. strobing an input port until a key press has been detected. 

This situation is referred to herein as a "channel sync error." Once a key press has been detected, processing proceeds to 

A bidirectional communication interface (not shown) 55 step S310. 

between the handheld remote 105 and the tuner devices may In step S310, the pressed key is tested to determine 

be implemented to prevent such channel sync errors from whether it is universal remote key, such as a volume control 

occurring. Of course, the tuner device must also be capable key, or a channel changing key. If the pressed key is a 

of bidirectional communication, but it is envisioned that universal remote key, then the key is processed in step S311. 

equipment manufacturers will incorporate such interfaces in 60 If the pressed key is not a universal remote key, then 

their future tuner devices. Details of implementing bidirec- processing continues to step S320, where a test is performed 

tional communication interfaces (e.g., a bidirectional IR to determine whether the pressed key is the event button 

interface) are well known. (referred to in FIG. 8 as the record time stamp key). 

When a bidirectional interface is available, and the hand- If the pressed key is the event button, it is processed in 

held remote 105 issues a command to a tuner device, the 65 step S321, where the remote stores the date, time, channel 

handheld remote can verify that the tuner device actually and current device in the time stamp table 280, thereby 
received the command by waiting for an acknowledgment creating an entry in that table. In the example depicted in 
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FIG. 7, ihe second entry indicates that the event button was computer 502 is connected to appropriate interface circuitry 

pressed while the user was watching channel 5 from the 501, either via an internal bus, or via a connector and a cable, 

cable box on Nov. 10, 1998 at 8:26 PM. Alternatively, in This interface circuitry 501 enables the computer 502 to 

embodiments where only one station is supported by the communicate with the IR transceiver 500. The connections 

system, the channel and current device information can be 5 and communication between the computer 502 and the IR 

omitted from the time stamp table, because those fields transceiver 500 can be implemented in any conventional 

would always contain the same information. Returning now manner. One example of a suitable device for performing 

to FIG. 8, the time stamp data for the event is displayed on mis ft, ncl i on ^ the Actisys IR220L. 

the LCD of ^ handheld remote. To commimication ^ mc homc computcr 50 2, 

If, in step S320, the pressed key * not tie event button 10 ^ ^ ^ remote at ^ IR tnnsceWt[ sm 

processing continues to step S330 where a test is performed CQmcci £ tQ ^ uter 502 ^ ^ to s thc 

to determine whether the pressed key is the transfer com- ' uu , 7" *" ^ y , „ : . " . . \ r a 

mand (referred to in FIG. 8 as the download time stamps butt0D ™ his handheld remote to transfer the Ume 

key). If it is a transfer command, the time stamps are stamps stored m the remote to the computer TTiese Ume 

transmitted to a computer receiver via the IR transceiver in stamps are received by the computer 502 via the interface 

step S331. If the pressed key is not a transfer command, 15 circuitry 501. 

processing continues in step S340. Although the transfer of time stamps from the remote to 

Step 340 is implemented only in embodiments that sup- the home computer described above only involves data 

port browsing directly from a handheld remote, which is an flowing in one direction (i.e., from the remote to the 

optional feature. Dedicated keys may be added to the remote computer), two-way communication may also be used to 

for this purpose. In step S340, the pressed key is tested to 20 implement handshaking or to improve the reliability of the 

determine whether it is a browser key. If it is, processing of data transmission, in any conventional manner. Two-way 

the browser key occurs in step S341, which preferably communication is also useful to initialize (or re-initialize) 

forwards the browser key directly to the computer. When the remote as to which brands of equipment are installed in 

browsing from the remote is not implemented, browsing can the user's home, as explained above. Optionally, two-way 

be accomplished in any traditional manner, such as using a 25 communication may be used to send messages to the user via 

mouse attached to the home computer. the LCD display on the remote. 

Finally, after the pressed key is processed, the program FIG. 11 shows the processing that occurs in the remote 

returns to step S300 to wait for the next key press. when data is sent from the remote to the computer using 

FIG. 9 describes the processing of universal remote keys. 3Q two-way communication. First, in step S 610, the time 

Processing begins in step S400. In step S410, if the pressed stamps stored in the time stamp table 280 (shown in FIG. 7) 

key is a power key for a tuner device (i.e., a key which turns are encoded as binary pulses. Then, in step S620, the user is 

on a tuner), then that key is processed in step S411. In step prompted to point the handheld remote at the computer's IR 

S411, the remote issues an IR command to turn on the transceiver. This prompt will appear on the LCD display of 

device, followed by an IR control command which changes 35 the handheld remote in embodiments that have such a 

the device's channel to the channel that is already stored in display. Next, in step S630, the binary pulses are transmitted 

the handheld remote's internal register. This channel setting to the computer via the IR transceiver. In step S640, the 

is obtained from the tuner device current state table 260 remote waits for acknowledgment of the time stamps that 

(shown in FIG. 5). This ensures that the variables in the were transmitted to the computer. In step S650, a test is 

handheld remote match the actual settings of the devices, 4Q performed to determine whether the transmitted data was 

forcing the devices to a known state. received correctly by the computer, preferably based on the 

If the pressed key is not a power key, processing continues acknowledgment received in step S640. If the transmitted 

in step S420, where a test is performed to determine whether time stamps were not received correctly by the computer, 

the pressed key is a channel change key. If it is a channel processing returns to step S620. If the data was received 

change key, the pressed key is processed in step S421. In 45 correctly, processing of this subroutine ends, 

step S421, the remote issues an IR command to change the FIG. 12 depicts the process that runs in the computer to 

channel on the currently selected device. It also stores the communicate with the handheld remote. Processing begins 

new channel setting in the tuner device current state table in step S700 where the computer waits for an IR commu- 

260 (shown in FIG. 5). nication signal to arrive from the handheld remote. When the 

If the pressed key is not a channel change key, then 50 communication signal arrives, a test is performed in step 

processing continues in step S430 where a test is performed S710 to determine whether the data was received correctly, 

to determine whether the pressed key is a display device If the data was not received correctly, the computer prompts 

change key. If the pressed key is a display device change the user in step S711, using a suitable display or audio 

key, it is processed in step S431. In step S431, the handheld message, to point the remote at the computer IR transceiver, 

remote issues an IR command to change the display device 55 If it is determined in step S710 that the data was received 

being displayed on the television. The new current display correctly, processing continues in step S720 to determine 

setting is also stored in the event variables table 270 (shown whether the received data represents time stamps. If the 

in FIG. 6). received data represents time stamps, the time stamps are 

If the pressed key is not a display device change key, then processed in step S721. 

processing continues to step S440 where the handheld 60 If the received data is not a time stamp, the received data 

remote issues an IR command corresponding to the key is tested to determine whether it is a browser key in step 

pressed. Keys that are processed in this step include all keys S730 (in embodiments that support browsing from the 

that are not power, channel change, or display change remote). If it is a browser key, it is processed in step S731. 

command keys (e.g., a volume change command). Finally, This processing would include forwarding the browser key 

this subroutine ends in step 450. 65 to a web browser running on the home computer, which then 

FIG. 10 depicts an example of a home computer setup communicates with the server via the Internet in a conven- 
suitable for use with the present invention. The home tional manner. 
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FIG 13 depicts how the time stamps are received and in the program that was being broadcast when the event 

processed by the computer. After the time stamps are button was pressed. The server can use the determined 

received in step S810, the received time stamps are tested in program time to index into a product database that identifies 

step S820 to determine whether the data was received the products that appear at each tune in the program, and 

correcdy If the data was not received correctly, the com- s thereby determine which products were being displayed at 

puter prompts the user, in step S821, to point the handheld the time the event button was pressed. This process of skew 

remote at the computer IR transceiver. If the data was compensation is described in greater detail below, 

received correctly, the time stamps that were encoded in the In addition to obtaining the information contained in the 

handheld remote are decoded in step S830. Those time time stamps, the server also must obtain information about 

stamps are then stored in a time stamp table in the computer the television setup in the user's home. Preferably, this is 

in step S840 accomplished by obtaining a copy of the service and device 

Next, in step S850, the computer will request permission configuration table 250 (shown in FIG. 4, hereinafter "the 

from the user to be^in processing an order and test for SDC table"). This js preferably accomplished by rar^fer- 

. r ~7 . & . y . , c | <ft Tf MrTT1 : cc ; rtn ; c nnt ring a copy of the SDC table from the home computer to the 

receipt of this] permission in step S860. If permissio^ * no t g PY fa from ^ 

granted (which might occur, for example, if the phone Ime 15 

used Optional customer information for each user may also 

subroutine ends m step S870. It permission is granted, order v j> ,4 in <u 0 „™ r 

processing is set up in step S861, in which case the computer be stored in the home computer, and transferred to the serve 

wiUestablishaconnecaontotheorderprocessmgserversite together with the time stamps. £^yely, to cus^u 

via the Internet in any conventional manner, and forward the M mfonnauon could be stored m the server in which case it 

time stamps and the SDC table information to the server. would not bo transmuted toge ther with the .mesta mps. The 

Preferably, the connection to the server site uses the hyper- customer ^formation could include a customer code that 

text transfer protocol (HTTP). Once a connection has been uniquely identifies each subscriber to 'be system and 

suitably established, the computer encrypts the time stamp op^V additional information about the sub- 

formation and the*SDC table and, optionally, information „ -riber. If «* SDC table ^f^^^^ 

related to the user (including, e.g., account information). " ™mber could also be used to access the customer s SDC 

This encrypted information is transmitted to the server via lable - 

the Internei, also in a conventional manner. Communication Once the time stamps and SDC table havebeen received 

with a server then continues as with any other website. by the server, the time stamps are processed. This processing 
Once the server receives the time stamp, the server must 30 involves additional informal which * . preferab y stored 

compensate for all time shifts that were introduced when the "> "Wes on the > server, including , the med a service tab e 905 

program was broadcast. This is accomplished by mapping shown m FIG. 15) and the broadcast report table 910 

each second of the broadcast, which occurs in real time, into (shown in Hu. lob). 

a second of the original uncut program. This process com- The media service table 905 indicates which network 
pensates for time shift and skews caused by, for example, 35 corresponds to a given channe for " »™*™»- For 

delays in starting a program, and commercials that are example, the first row of data in the media service table 905 

interspersed with the program. Scales that channel 20 on service 3888 corresponds to 

nG.Mshowsanexampleofthismappmg.The-program USAnetwo*s . Tlie se^nd row mdicates ^ c hannell4on 

time" time line 900 shows the time with respect to the semce 3888 corresponds to HB01. The service ffi field in 
original program. The "broadcast time" time line 901 shows 40 this table indicates the service provider (e.g MuhtiUn 

, & F ; fr 4 . , P( . „, rt(T r,rr, Cable! and corresponds to the service ID field in the bUL 

the actual times that each segment of the prog am wa ^ P en oo ^ ^ 

broadcast. Ordinarily, each moment :ol f^^^ b l c also nchides a location code whi^h can be used to store 

map into only one corresponding moment or broadcast time. a *™ 

In "he illustrated example, minutes 20 through 32 of the information about a tune zone. 

program were broadcasted from 8:22 PM to 8:34 PM on Jan. 45 A program production table 909 (shown in FIG. 16A) 

1 1999. After this 12 minute segment, 3 minutes of com- holds information that indicates which company produced 

mercials were broadcasted from 8:34 PM to 8:37 PM. Then, each program, and a unique content ID that identifies the 

the next 8 minute segment of the program was broadcasted program. Some programs may be supported by the server 

from 8:37 PM to 8:45 PM. Next, a 4 minute commercial was one year, but not supported m another year. When only 
broadcasted from 8:45 PM to 8:49 PM. Minutes 40-44 of 50 supported programs appear in the table 909 the program 

the program were never properly broadcast due to technical production table can be used to indicate which programs are 

difficulties, which is indicated in the broadcast time line supported by the server. Alternatively, a field can be added 

from 8-49-8-53 PM. The next section of the program to each entry in the program production table 909 to mdicate 

(beginning at minute 44), was broadcasted starting at 8:53 whether each program is supported. 
p M 55 The fourth entry in the program production table 909 is an 

The process of mapping broadcast time to program time example of using the present invention to commercialize 

is referred to as skew compensation. This skew compensa- products that appear within commercials and it shows how 

tion process can be implemented for programs of any length, commercials can be treated like any other program For 

including, for example, a balf-hour sitcom episode, a two example, if the announcer in commercial happens to be 
hour movie, (which, when commercials are added, might 60 wearing a certain hat, and the .user presses the ; event button, 

take 2.5 hours to broadcast), or even a three hour movie, this event would be processed just like event button presses 

broadcasted in two or more installments on different days. In that occur during ordinary programs. Instead of identifying 

the latter case, the skew compensation process maps two or a program that corresponds to the time stamp, however, the 

more broadcast time lines (one for each day), onto the system would identify the commercial that corresponds to 
original three hours of program time. The server uses this 65 the time stamp. 

map to convert the real time information from each time A broadcast report table 910 (shown in FIG. 16B) is 

stamp into a corresponding program time, which is the time provided to the server for each individual network supported 
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by the server. Alternatively, although not so depicted, a ing the signal to the cable box in the user's home. In the 

single broadcast report table may be used for all networks, example under consideration, this information would be 

provided that a network identifier entry is added to the table. extracted from the first row of the service and device 

The broadcast report table 910 is used by the server to configuration table, which identifies Manhattan Cable, 

determine (1) which program the user was watching when 5 which has an associated service ID code of 3888. In embodi- 

he pressed the event button; and (2) to provide synchroni- ments where only one station is supported by the system, 

zation information which can be used to perform skew this step can be omitted because the service is always 

compensation. constant. 

The first row of data in the illustrated broadcast report Next, in step S1010, the server determines the network 

table 910 indicates that a segment of a program began io that was being watched by the user at the time the event 

broadcasting at 8:22 PM and stopped broadcasting at 8:34 button was pressed. This is accomplished by using the 

PM, and that the content ID of that program is 2039312. The service ID determined in step S1000 and the channel 

content ID uniquely identifies a particular program (e.g., extracted from the time stamp as indexes into the media 

episode #12 of Melrose Place, or an edited-for-TV version service table 905 (shown in FIG. 15). In the example under 

of The Shawshank Redemption, corresponding to the entry 15 consideration, the service ID of 3888 and the channel 5 

in the program production table 909). The synchronization appear in the third row of the media service table, which 

information in tie two right-hand columns of the broadcast indicates that the network is Fox USA. In embodiments 

report table 910 indicates that at 8:25 PM, the 23 rd minute where only one station is supported by the system, this step 

of that program was broadcast. can also be omitted because the network is always constant. 

The third row of broadcast report table 910 indicates that 20 Preferably, the media service table also includes a location 

the same program (i.e., program 2039312) started up again code which indicates the location of the user who pressed the 

at 8:37 PM and continued until 8:45 PM. The synchroniza- event button. This location code is used to compensate for 

tion entries for this row coincides with the start of the users located in different time zones. Optionally, to handle 

segment, since the fade-in time and the broadcast sync time cases in which two different networks broadcast over the 

are the same (i.e., 8:37 PM, which corresponds to minute 32 25 same channel at different times, a time entry (not shown) can 

of the program). also be included in the media service table. If this is done, 

Synchronization entries may be performed semi- the time from the time stamp under consideration would also 

automatically by, for example, having an operator in the be used as an index into the media service table, 

broadcasting studio enter the synchronization data by noting In step S1020, the server determines the particular pro- 

that at 8:25 PM, the 23 rd minute of a particular show (such 30 gram that the user was watching when the event button was 

as episode No. 12 of Melrose Place) was airing. This pressed. This is accomplished by accessing the broadcast 

example is illustrated in the first row of data in the broadcast report table 910 (shown in FIG. 16B) for the particular 

report table 910. After the data for a given show (or portion network determined in step SI 010, and using the time from 

thereof) is complete, the broadcast report table entries can be the time stamp as an index into that broadcast report table 

sent to the server, via, for example, a modem connection. 35 910. Alternatively, in cases where multiple networks are 

More preferably, the synchronization is performed auto- included in a single broadcast report table with an entry to 

matically using a continuous communication link to send specify the network, both the network determined in step 

each entry from a cooperating broadcaster to the server S1010 and the time from the time stamp are used as indexes 

individually. For example, each entry may be sent at the start 4Q into the broadcast report table. 

of each broadcasted segment. In the example illustrated in In the example under consideration, the broadcast report 

the third row of the broadcast table 910, the broadcaster table 910 is assumed to be the broadcast report table from 

transmits the program synchronization time of 32 minutes at the Fox USA network, and the time extracted from the time 

8:37 PM, which is the time that they fade in the segment that stamp is 8:26. The server uses this extracted time as an index 

starts at the 32 nd minute of the program. When this 45 into Fox's broadcast report table 910, and searches for 

arrangement is used, a single entry can be used to represent entries that correspond to the extracted time. Preferably, this 

both the fade in time and the broadcast synchronization time, is accomplishing by checking if the extracted time is later 

because they will always be the same. than the fade-in time and earlier than the fade-out time. 

Optionally, the fade-out time can be eliminated from the Here, the extracted time of 8:26 falls between the fade-in 

broadcast report table 910 by using the fade-in time from the 50 time and the fade-out time of the first row of broadcast report 

next entry table 910 This P oints to me program with a content ID of 

FIG. 17 is a flow chart that shows how the server ^039312, which uniquely identifies the program that was 

processes each time stamp to determine the time within the *W watched when the event button was pressed, 

program when the event button was pressed, based on the In cases when the identified program is not supported by 

time stamp itself, the SDC table, the media service table, and 55 *e system, an appropriate message can be returned to the 

the broadcast report table. This will be explained using the user via the Internet. 

example of the second entry in the time stamp table 280 Next, in step S1030, the server performs skew compen- 

(shown in FIG. 7). As explained above, copies of this table sation to determine the particular time within the program 

and the SDC table have been sent to the server. when the event button was pressed. This is accomplished 

First, in step S1000, the server determines the particular 60 using the synchronization information extracted from the 

service that is associated with the time stamp. To accomplish broadcast report table 910. The preferred algorithm for 

this, the server extracts the tuner device for a given time determining the program time (i.e., the time within the 

stamp from the time stamp table 280 (shown in FIG. 7). In program) is implemented using the following equations: 

the example under consideration, this would be the cable ^ pts=pst+BTS-bst 

box. Then, the server uses this extracted information as an 65 

index into the service and device configuration table 250 2 ) psb=pst + bts-bst 

(shown in FIG. 4) to determine the service that was provid- 3) pse*pst+bts-bst 
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where PST is the program synchronization time, PS B is the user's home computer via the Internet in any conventional 

program segment begin time, PSE is the program segment manner to enhance the presentation of the assortment of 

end time, PTS is the time stamp time measured in program products. 

time, BTS is the time stamp time measured in broadcast By implementing this option, the system can easily verify 

time, BST is the broadcast synchronization time, FIT is the S that the computed program and time correspond to the 

fade-in lime, and FOT is the fade-out time, program and time that was actually being watched by the 

When the broadcast report table 910 is provided auto- user when the event button was pressed. This would be 

matically at the start of each broadcast segment, as discussed useful, for example, when a channel sync error has occurred, 

above, the broadcast synchronization time and the fade-in e.g., when a user presses a channel-change button on the 

time will be the same, which simplifies equation #2. 10 remote, but the remote is not aimed at the television. In that 

Continuing with the example under consideration where case, the channel being tracked in the remote would not 

the broadcast time stamp is 8:26, and plugging the relevant match the channel actually being watched. By displaying a 

times into equation 1, the result is: still image or video clip from the program, the system can 

verify the computed program and time by querying the user. 

PTS-0:23+8:26-8^5 15 If ^ enor ^ ^ Gi&ci ^ fa sys t e m can then query the user to 

Solving this equation results in a program time of 0:24. determine which program was actually being watched, and 

This means that when the event button was pressed at 8:26 then reprocess the time stamp for that program. 

PM, the 24 th minute of the program was being displayed. Alternatively, each product may be presented to the user 

It should be noted that while the above example uses one individually, in sequence. As yet another alternative, all of 

minute increments, using smaller increments (e.g., one 20 the products for all of the time stamps may be presented to 

second, or 0.1 seconds) is preferable. the user simultaneously. Numerous other presentation 

After determining the time within the particular program approaches will be apparent to persons skilled in the relevant 

when the event button was pressed, the products that were art. 

being displayed at that time can be determined by referenc- Finally, the server continues to operate as a conventional 

ing the product display table 920 (shown in FIG. 19). 25 web server, using browsing commands from the home 

Preferably, the product display table is created before computer's web browser (which is operated using either the 

broadcasting the program, and inputted into the server. A remote or the computer keyboard, as described above) to 

product display table can be made for any given program by provide information and promote the products in the assort - 

watching the uncut program and waiting to see whether ment. 

products supported by the system appear. When those prod- 30 In the preferred embodiments described above, each time 

ucts appear, a product description is entered into the product the event button is pressed, a corresponding time stamp is 

display table, together with an entry for the appear time (i.e. stored in the handheld remote 105. In these embodiments, 

the time that the product appeared), and a disappear time. Of the user's home computer 106 may remain off while the user 

course, alternative data structures may be used, such as watches television, and only needs to be turned on when the 

using a product code instead of the product name, or using 35 user wishes to transmit the time stamps into the home 

the duration of appearance instead of the disappear time. computer 106 and upload them to the server. 

An example of this process is illustrated in FIG. 18, which If a home computer 106 is available during the entire TV 

is self-explanatory. An example of the resulting product watching session, an alternative preferred embodiment 

display table 920 is illustrated in FIG. 19. Each time a (hereinafter "the second preferred embodiment") may be 

product appears, a new entry is generated in the product 40 implemented. This second preferred embodiment relies on 

display table 920. The content ID column of the product the home computer 106 to keep track of which tuner device 

display table 920 identifies a particular program (e.g., epi- and channel is being watched by the user, to keep track of 

sode No. 12 of Melrose Place). This process continues until the current time, and to accumulate the time stamps. In this 

the program is finished. The product display table 920 may second preferred embodiment, the handheld remote 105 is 

be inputted into the server in any conventional manner. 45 freed from performing these tasks. To accomplish this, the 

Additional information about the supported products are handheld remote 105 communicates with the home com- 

also inputted by the server in the product table 930 (shown puter 106 each time the event button, a channel-change 

in FIG. 20). The entries in this table are self explanatory. button, a display device change button, or a power button on 

Any suitable alternative or additional information may also the handheld remote 105 is pressed, 

be included in the product table 930, including information so The overall system block diagram for this second pre- 

coramonly used in computer-assisted marketing systems. ferred embodiment is similar to the configuration shown in 

Returning now to FIG. 17, once the program time has FIG. 1, except that a bidirectional radio frequency (RF) link 

been determined in step SI 030, control passes to step 1040 is preferably used to communicate between the handheld 

where the server searches the product display table for remote 105 and the home computer 106 instead of the 

entries in which the appear time comes before the time 55 infrared interface (IRVO) shown in FIG. 1. The RF link may 

stamp at issue, and the disappear time comes after the time be accomplished using any of a variety of techniques well 

stamp at issue (allowing for a time tolerance). The server known to those skilled in the art, including, for example, 

selects the products that meet these criteria and presents the encoding digital data using frequency shift keying (FSK) or 

resulting assortment of products to the user. An example of phase shift keying (PSK) before it is transmitted. Examples 

this process is illustrated in FIG. 21, which is self- 60 of suitable RF communication standards include Lucent 

explanatory. Technologies* Wavelan wireless network standard and the 

The products can be presented to the user in batches, with Bluetooth RF network standard, 

each batch corresponding to a particular time stamp. Using an RF link enables the handheld remote 105 to 

Optionally, the server may retrieve a still-frame image or communicate with the home computer 106 without requiring 

video clip from the program that recreates the entire image 65 an aiming operation (such as pointing the handheld remote 

that appeared on the user's television of the time when the 105 at the home computer). Because an RF link is available, 

event button was pressed. This image can be transferred to initialization of the handheld remote 105 is preferably 
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accomplished via RF commands that are transmitted to the 
handheld remote 105 from the home computer 106 via the 
RF transceiver. 

FIG. 23 is a block diagram of a handheld remote 105 for 
use with this second preferred embodiment. The hardware 
configuration of the handheld remote 105 of this embodi- 
ment is similar to the arrangement of the first preferred 
embodiment shown in FIG. 2, except that an RF transceiver 
1208 is added for establishing a communication link with 
the home computer 106a. In addition, a transmit-only IR 
transmitter 1207 is used in place of the IR transceiver (Ref. 
No. 207 in FIG. 2), unless a bidirectional communication 
link is implemented between the handheld remote 105 and 
the tuner devices (in which case a transceiver would still be 
required). 

The handheld remote 105 of this embodiment includes the 
conventional buttons found on traditional universal remotes 
plus one additional "event" button. A "transfer" button is not 
needed in this embodiment. 

FIG. 24 is a flowchart depicting the operation of the 
handheld remote 105 of this second preferred embodiment 
in the context of user-initiated controls. Preferably, the 
handheld remote 105 in this embodiment acts as a "dumb" 
input device for the home computer 106, and as much 
intelligence as possible is shifted out of the handheld remote 
105 and into the home computer 106. In step S1300, the 
remote remains in a quiescent state until a key on the 
handheld remote is pressed or until an RF command is 
received from the home computer 106. 

As soon as a key-press is detected, processing proceeds to 
step S1310, where a test is performed to determine whether 
the key is a member of the following set of "special" keys: 
power keys, channel change keys, display device change 
keys, and the event key. If the pressed key is a member of 
that set, the key is referred to herein as "special". 

If the pressed key is not "special" (e.g., a volume change 
key), processing proceeds to step S1311, where the infrared 
code corresponding to the pressed key is transmitted via the 
IR transmitter 1207. 

If the pressed key is "special", processing proceeds to step 
S1320, where the key-press is reported to the home com- 
puter 106 via the RF transceiver 1208 (shown in FIG. 23). 
Next, the handheld remote 105 waits for instructions from 
the home computer 106, which are received in step S1330. 
These instructions inform the remote which, if any, infrared 
codes should be transmitted via the infrared transmitter 
1207. Preferably, these codes will correspond to the keys 
that were originally pressed on the handheld remote 105. For 
example, if the user originally pressed the "channel up" key 
on the handheld remote 105, the instructions received from 
the home computer 106 will instruct the handheld remote 

105 to generate the appropriate infrared codes to change the 
channel on the currently selected tuner device. The handheld 
remote 105 will not generate any IR code in response to a 
null instruction received from the home computer 106. In 
step S1340, the handheld remote 105 transmits the infrared 
code or codes corresponding to the instructions received 
from the home computer 106 via the infrared transmitter 
1207. Optionally, a bidirectional interface may be imple- 
mented between the handheld remote 105 and the tuner 
device when IR codes are transmitted, as discussed above in 
connection with the first embodiment. Control then returns 
to step S1300, where the handheld remote 105 will remain 
in a quiescent state until the next key is pressed or until an 
RF command is received. 

In this second preferred embodiment, the home computer 

106 tracks the date and time, as well as the tuner device and 



channel that are being watched (based on channel and tuner 
change information received from the handheld remote 
105). The home computer 106 preferably includes a set of 
memory locations to store the current state of the user's TV 
5 viewing setup in order to perform this tracking. Preferably, 
each of the service and device configuration 250, the tuner 
device current slate table 260, the event variables table 270, 
and the time stamp table 280 (shown, respectively, in FIGS. 
4-7) are stored in the home computer 106 in this second 
10 preferred embodiment. Preferably, the home computer 106 
also maintains a buffer for temporarily holding the instruc- 
tions that are to be sent to the handheld remote 105. 

FIG. 25 is a flowchart of a program that runs in the home 
computer 106 in this second preferred embodiment and 
responds to button-presses that occur on the handheld 
remote 105. Processing for this program begins in step 
S1400, where the home computer 106 waits for a transmis- 
sion from the handheld remote 105 via an RF transceiver 
(not shown) connected to the home computer 106. 
Preferably, each transmission from the handheld remote 105 
to the home computer 106 will correspond to a single 
key-press on the handheld remote 105. When a transmission 
from the handheld remote 105 is received, the date and time 
of arrival are extracted from the computer's real time clock 
in any conventional manner and stored in the event variables 
table 270 (FIG. 6) in step S1405. 

Next, tests are performed to determine what type of key 
was pressed. First, in step S1410, a test is performed to 
determine whether the pressed key is a power key for a tuner 
device. If the pressed key is a power key, processing 
continues in step S1411, where the home computer 106 
loads the buffer with instructions to power on the selected 
device, and also loads the buffer with instructions to change 
the channel on that device to a pre-stored setting. This 
pre-stored setting is preferably retrieved from the device 
current state table 260 (FIG. 5). When these instructions are 
received by the handheld remote 105, the handheld remote 
105 will generate appropriate IR codes to turn on the 
selected device and to initialize that device's channel setting 
to match the pre-stored setting. 

If the test in step S1410 indicates that the pressed key is 
not a power key, processing continues at step S1420, where 
a test is performed to determine whether the pressed key is 
a channel change key. If the key is a channel change key, 
processing continues in step S1421 where the home com- 
puter 106 determines the new channel setting and updates 
the tuner device current state table 260 (FIG. 5). This may 
be accomplished by either incrementing or decrementing the 
previous channel setting (preferably retrieved from the tuner 
device current state table 260) when a channel-up or a 
channel-down command is received, or by jumping to a new 
channel when direct entry numeric keys are pressed on the 
handheld remote 105. In addition, the home computer 106 
will load the instruction buffer with instructions to change 
the device channel on the currently selected tuner device. 
When these instructions are received by the handheld remote 
105, the handheld remote 105 will generate appropriate IR 
codes to cause the channel to change on the currently 
selected tuner device. 

If the test performed in step S1420 indicates that the 
pressed key is not a channel change key, processing contin- 
ues in step S1430, where a test is performed to determine 
whether the pressed key is a display device change key. If 
the key is a display device change key, processing continues 
in step S1431, where the home computer 106 will store the 
new current display device setting by modifying the contents 
of the event variables table 270 (FIG. 6). In addition, the 
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home computer 106 will load the instruction buffer with 
instructions to change the display device. When these 
instructions are received by the handheld remote 105, the 
handheld remote 105 will generate appropriate IR codes to 
select the desired display device. 

If the test performed in step S1430 indicates that the 
pressed key is not a display device change key, processing 
continues in step S1440, where a test is performed to 
determine whether the pressed key is the event button. If the 
pressed key is the event button, processing continues at step 
S1441, where the home computer 106 adds an entry to the 
time stamp table 280. Preferably, this is accomplished by 
extracting the date, time, and current display device from the 
event variable table 270 and the channel from the tuner 
device current state tabic 260 (FIGS. 5 and 6), and storing 
the extracted information as an entry in the time stamp table 
280 (FIG. 7). Each time the event button is pressed, an 
additional entry is added to the time stamp table 280. 

When the event button is pressed, a null instruction is 
preferably added to the instruction buffer. In response to 
receipt of this null instruction, the handheld remote 105 will 
not generate any IR codes. If the test performed in step 
S1440 indicates that the pressed key is not the event button, 
the processing routine ends at step SI 460 without perform- 
ing any additional steps. 

After the instruction buffer is loaded in any of steps 
S1411, S1421, S1431, and S1441, processing proceeds at 
step S1450, where the contents of the instruction buffer are 
transmitted via the RF transceiver 1208 (shown in FIG, 23) 
to the handheld remote 105. 

Once the user has finished generating new events, the user 
informs the home computer 106 that he has completed 
entering events in any conventional manner (e.g., by press- 
ing a function key on the computer's keyboard, or by 
clicking a mouse on an appropriate icon or button). At this 
point, the home computer 106 will attempt to establish a 
connection to the order processing server site 107 via the 
Internet, and forward the collected time stamps and the SDC 
table 250 (FIG. 4) to the server 107 (as in the first 
embodiment). 

Once the time stamps have been forwarded to the server 
107 in this second preferred embodiment, processing in the 
home computer 106 then continues in the same manner as it 
did in the first embodiment. The processing that occurs in the 
server 107 in this embodiment would then be identical to the 
processing in the first embodiment described above, because 
the server receives the time stamps from the home computer 
106 in a batch (and the manner in which those time stamps 
were initially collected does not impact the processing 
performed in the server). 

In a first variation of this second preferred embodiment, 
instead of waiting for a response from the home computer 
106 before issuing the IR command, the handheld remote 
105 may determine the appropriate infrared codes by itself 
based on the key that was pressed, and immediately transmit 
the corresponding IR codes via the infrared transmitter 
1207. The communication with the PC (e.g., via RF) may 
then be performed after the IR codes have been transmitted, 
or simultaneously therewith. This variation may be imple- 
mented when either a bidirectional interface or a unidirec- 
tional interface is used to communicate between the hand- 
held remote 105 and the tuner devices. 

In a second variation of this second preferred 
embodiment, when a real time connection with the Internet 
is available, the time stamps may be sent to the server 107 
one at a time, as soon as each one is created in the home 
computer 106. The server could then return information to 
the user while the user is still watching television, if the user 
so desires. 



If a real-time connection with the Internet is always 
available, yet another preferred embodiment (hereinafter 
"the third preferred embodiment'*) may be implemented. In 
this embodiment, the home computer 106 acts as a simple 
conduit that forwards information from the handheld remote 
105 to the server 107 and from the server 107 to the 
handheld remote 105. All the processing implemented in the 
home computer 106 in the second preferred embodiment is 
shifted to the server 107, so that the server keeps track of the 
date and time, as well as the tuner device and channel that 
are being watched. Preferably, this is accomplished by 
storing the service and device configuration table 250, the 
tuner device current state table 260, the event variables table 
270, and the time stamp table 280 (shown in FIGS. 4-7) at 
the server 107. 

In this third preferred embodiment, each time any of the 
"special" keys on the handheld remote 105 is pressed, the 
handheld remote 105 immediately transmits information to 
the home computer 106 indicating which key was pressed. 
The home computer 106 then immediately forwards this 
information to the server 107, preferably via the Internet. 
The server 107 then implements the same process that was 
implemented by the home computer 106 in the second 
preferred embodiment (steps S140O-S1460, described 
above in connection with FIG. 25). The only change to that 
process is that in step S1450, instead of transmitting the 
instructions direcdy to the handheld remote 105 (as in the 
second preferred embodiment), the transmission to the hand- 
held remote 105 in the third preferred embodiment is 
accomplished by transmitting the instructions to the home 
computer 106, and having the home computer 106 forward 
the instructions to the handheld remote 105. 

Once the instructions are received by the handheld remote 
105, the handheld remote 105 will transmit the IR codes that 
correspond to the received instructions. It is noteworthy that 
the hardware and software in the handheld remote 105 of 
this third preferred embodiment are preferably identical to 
the hardware and software in the handheld remote 105 of the 
second preferred embodiment (as described in connection 
with FIGS. 23 and 24). 

The major difference between the second and third pre- 
ferred embodiments is that the processing performed in the 
home computer 106 in the second preferred embodiment is 
shifted to the server 107 in the third preferred embodiment. 
Because of this shift, the server 107 in this third preferred 
embodiment does not actually receive completed time 
stamps from the home computer 106. Instead, the time 
information for each time stamp is determined by the server 
107 based on a time code provided by the data transmission 
so protocol. For example, when the http protocol is used, the 
time that each data packet was sent out onto the Internet can 
be determined by examining the data packet, and does not 
depend on when that data packet actually reaches its desti- 
nation. Because the server 107 keeps track of the time in this 
third preferred embodiment based on the transmission 
protocol, the home computer 106 and the handheld remote 
105 do not have to keep track of the time. 

For example, if a user presses the event button on Feb. 1, 
1999 at 9:03:05 PM, the key-press would be immediately 
transmitted from the handheld remote 105 to the home 
computer 106 via RF, and one or more packets of corre- 
sponding data would be immediately transmitted from the 
home computer 106 to the server 107 via the Internet. No 
time data is explicitly included in the transmitted data itself. 
The packets of data will usually arrive at the server within 
a few seconds of 9:03:05 PM. The server 107 then extracts 
the time and date from the http protocol information asso- 
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ciated with the received packet, reads the current display 
device from the event variables table 270, and reads the 
current channel for that display device from the tuner device 
current state table 260. The server 107 then generates a time 
stamp using this time, dale, device, and channel information, s 
and stores the time stamp in the time stamp table 280. 

In contrast to the previously-described embodiments in 
which the time stamps are assembled in the home computer 
106, the time stamps in this third preferred embodiment are 
generated inside the server 107 individually, each in 
response to a single press of the event button. When the 
timestamps are generated in the server one at time in this 
manner (or when they arrive one at a time in the second 
variation of the second embodiment), the server 107 may 
optionally send information back to the user via the Internet 
while the user is still watching TV. The viewer would then 15 
be free to direct his attention to this information, if he so 
chooses. Alternatively, the server 107 in this third embodi- 
ment may accumulate the generated time stamps, and wait 
for the user to initiate an inquiry. Upon receipt of this 
inquiry, processing of the time stamps may proceed in the 20 
same manner described above in connection with the first 
and second preferred embodiments. 

While the first, second, and third preferred embodiments 
described above each disclose a specific allocation of data 
storage and processing tasks among the handheld remote 25 
105, the home computer 106, and the server 107, various 
alternative allocations can also be implemented. For 
example, a hybrid of the first and second embodiments may 
be implemented by linking the handheld remote 105 to the 
home computer 106 using RF communications that are 30 
activated each time a "special'* key is pressed, tracking the 
channel in the handheld remote 105, and determining the 
time and date in the home computer 106 by noting the time 
that each communication is received from the handheld 
remote 105. Numerous alternative allocations can be readily 35 
envisioned. 

It is to be understood that the present invention is not 
limited to the specific embodiments described above, and 
that various changes and modifications can be effected 
without departing from the scope or spirit of the present 40 
invention. One such modification is depicted in FIG. 22 
which shows implementing the present invention in an 
operating system application space on a computer, a TV 
set-top box, or a digital TV. In this configuration, the 
functions of the television, the handheld remote, and the 45 
Home Computer are all combined into a single device. 
Similarly, the present invention can also be implemented in 
another devices that supports Internet communications. 
Another possible modification would be to implement the 
system to support only one broadcaster, and simplifying the 50 
processing performed in the server accordingly. 

Another modification would be to apply the present 
invention to other types of broadcasted video information, 
(including, for example, Internet broadcasts), or even to 
audio broadcasts (including, for example, FM radio). Yet 55 
another modification would be to send customer specified 
information from the computer to the remote during data 
transfer sessions. This information could include, for 
example, birthday and anniversary reminders that are ini- 
tially entered at the home computer. At the appropriate time 60 
(e.g., five days before the birthday), the remote can be 
programmed to display an appropriate message on the LCD 
display. Still another modification would be to distribute the 
various functions among different locations, by, for 
example, having the broadcaster perform some of the server 65 
processes described above. Numerous other alternative 
embodiments can be readily envisioned. 



We claim: 

1. A method of commercializing products that are present 
in a television broadcast of a program, the method compris- 
ing the steps of: 

inputting product information that identifies a plurality of 
products which are present in the program and a time 
of presence within the program for each of the prod- 
ucts; 

inputting skew information that identifies a correspon- 
dence between an actual time of broadcast and a 
relative time within the program for at least one seg- 
ment of the program; 

inputting a signal that was generated in response to an 
indication of interest made by a user while watching the 
television broadcast; 

determining a time-of-interest corresponding to the indi- 
cation of interest made by the user based on a time of 
arrival of the signal inputted in said signal inputting 
step; 

identifying a specific portion of the program that was 
being broadcast at the time-of-interest based on the 
time-of-interest and the inputted skew information; 

determining an assortment of products that were present 
in the television broadcast at the time-of-interest based 
on the specific portion identified in said identifying step 
and the inputted product information; and 

presenting the determined assortment of products to the 
user. 

2. The method of claim 1, wherein the television broad- 
cast includes a plurality of programs that are broadcasted in 
turn, and the step of identifying a specific portion of the 
program comprises the steps of: 

identifying one program selected from the plurality of 
programs based on the time-of-interest and inputted 
broadcast information that identifies when the one 
program was broadcasted; and 

identifying a specific portion of the one program that was 
being broadcast at the time-of-interest based on the 
time-of-interest and the inputted skew information. 

3. The method of claim 1, further comprising the steps of: 
receiving an indication from the user selecting a product 

from the presented assortment; and 
providing, to the user, information relating to the selected 
product. 

4. The method of claim 3, wherein said step of providing 
information to the user comprises at least one of: offering the 
selected product for sale, providing information about the 
selected product, providing a referral to a seller of the 
selected product, and providing a link to a seller of the 
selected product. 

5. The method of claim 1, wherein the product informa- 
tion includes, for each of the products, an appear time and 
a disappear time within the program. 

6. The method of claim 1, wherein the skew information 
is inputted from a television station during the television 
broadcast of the program. 

7. The method of claim 1, wherein the product informa- 
tion is inputted from a database prior to the television 
broadcast of the program. 

8. The method of claim 1, wherein, in said signal inputting 
step, the signal is inputted via the Internet, and 

wherein, in said presenting step, the determined assort- 
ment of products are presented to the user via the 
Internet. 

9. The method of claim 1, wherein said step of presenting 
the determined assortment of products to the user comprises 
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iransmitting to the user, via the Internet, an image of a scene 
that was being broadcast at the time selected by the user. 

10. A method of commercializing products that are 
present in a plurality of simultaneous television broadcasts 
of a plurality of programs over a plurality of channels, the 5 
method comprising the steps of: 

inputting, for each of the programs respectively, product 
information that identifies a plurality of products which 
are present in the respective program and a time of 1Q 
presence within the respective program for each of the 
products; 

inputting, for each of the programs respectively, skew 
information that identifies a correspondence between 
an actual time of broadcast and a relative time within 15 
the respective program for at least one segment of the 
respective program; 

inputting an event signal that was generated in response to 
an indication of interest made by a user while watching 
the television broadcast; 

determining a time-of-interest corresponding to the indi- 
cation of interest made by the user based on a time of 
arrival of the event signal inputted in said event signal 
inputting step; 25 

inputting a channel-change signal that was generated in 
response to a channel change command made by the 
user; 

determining a channel based on the channel change signal 3Q 
inputted in said channel-change signal inputting step; 

identifying a specific portion of a specific program that 
was being watched by the user at the time-of interest 
based on the time of interest, the channel determined in 
said channel determining step, and the inputted skew 35 
information; 

determining an assortment of products that were present 
in the specific program at the time-of-interest based on 
the specific portion identified in the identifying step and 
the inputted product information; and 40 

presenting the determined assortment of products to the 
user. 

11. The method of claim 10, wherein each of the televi- 
sion broadcasts includes a plurality of programs thai are 45 
broadcasted in turn, and the step of identifying a specific 
portion of the program comprises the steps of: 

identifying a broadcaster based on the channel determined 
in said channel-determining step; 

identifying one program selected from the plurality of 50 
programs for the identified broadcaster, based on the 
time-of-interest and inputted broadcast information, 
wherein the inputted broadcast information identifies 
when the one program was broadcasted by the identi- 
fied broadcaster; and 55 

identifying a specific portion of the one program that was 
being broadcast at the time-of-interest based on the 
time-of-interest and the inputted skew information. 
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12. The method of claim 10, further comprising the steps 

of: 

receiving an indication from the user selecting a product 

from the presented assortment; and 
providing, to the user, information relating to the selected 

product. 

13. The method of claim 12, wherein said step of pro- 
viding information to the user comprises at least one of: 
offering the selected product for sale, providing information 
about the selected product, providing a referral to a seller of 
the selected product, and providing a link to a seller of the 
selected product. 

14. The method of claim 10, wherein the product infor- 
mation includes, for each of the products, an appear time and 
a disappear time within the program. 

15. The method of claim 10, wherein the skew informa- 
tion is inputted from respective television stations during 
respective television broadcasts of respective programs. 

16. The method of claim 10, wherein the product infor- 
mation is inputted from a database prior to the television 
broadcast of each program. 

17. The method of claim 10, wherein the event signal is 
inputted via the Internet in said event signal inputting step, 

the channel change signal is inputted via the Internet in 
said channel-change signal inputting step, and 

the determined assortment of products are presented to the 
user via the Internet in said presenting step, 

18. The method of claim 10, wherein said step of pre- 
senting the determined assortment of products to the user 
comprises transmitting to the user, via the Internet, an image 
of a scene that was being watched by the user at the time 
selected by the user. 

19. A method of commercializing products that are 
present in a plurality of simultaneous television broadcasts 
of a plurality of programs over a plurality of channels, the 
method comprising the steps of: 

inputting, for each of the programs respectively, broadcast 
and product information that identifies a plurality of 
products which are present in the respective broadcast 
and a time of presence within the respective broadcast 
for each of the products; 

inputting an event signal that was generated in response to 
an indication of interest made by a user while watching 
any of the television broadcasts; 

inputting a channel-change signal that was generated in 
response to a channel -change command made by the 
user; 

determining an assortment of products that were present 
in a specific program that was being watched by the 
user at the time the user made the indication of interest 
based on the event signal, the channel-change signal, 
and the inputted broadcast and product information; 
and 

presenting the determined assortment of products to the 
user. 

* * * * * 
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